Face identification device

ABSTRACT

A method for identifying a person includes the steps of detecting the face of a person in an input image, determining the reliability of each feature value from the input image, and obtaining a plurality of feature values from the detected face. Based on the feature values obtained from the detected face, the feature values stored on a storage unit, and the reliability of each feature value, an identification result according to the detected face is decided. A device includes the components for implementing said method.

BACKGROUND OF THE RELATED ART

1. Field of the Invention

The present invention relates to a device and a method for identifying ahuman face.

2. Description of the Related Art

A conventional method for identifying a person (i.e. determining whoseface is included) involves comparing registered information, such asfeature values and performing a comparison between this information anda identification image. However, if conditions are not suitable foridentification, that is, if a subject wears an article covering all or apart of the face such as sunglasses or a mask (hereinafter referred toas a “worn article”), or the lighting environment is too dark or toobright, or the subject faces sideways, the identification accuracydecreases.

In order to solve such a problem, it has been proposed that faceidentification devices corresponding to various face directions beprepared, that the face direction be detected in an identificationimage, and that a suitable face identification device be used (see toPose Invariant Face Recognition, by F. J. Huang, Z. Zhou, H. Zhang, andT. Chen, Proceedings of the 4^(th) IEEE International Conference onAutomatic Face and Gesture Recognition, Grenoble, France, 2000, pp.245-250.) Further, Laid-Open Japanese Patent Application No. 2000-215308proposes providing an illumination sensor that senses the lightintensity at the time of photography and also proposes that algorithmsfor extracting feature values and algorithms for identification areselected based on the light intensity previously determined. As such,for a plurality of functions relating to extraction of feature valuesand identification provided as mentioned above, image registration,algorithm tuning, studying of various parameters etc., must be performedfor each of the functions.

Further Japanese Patent Publication No. 2923894, Laid-Open JapanesePatent Application No. 2002-015311 and 2003-132339 all discuss methodsfor maintaining identification accuracy in which an image or featurevalue determined when there are no unfavorable conditions is used torestore an image in which there are unfavorable conditions. However,with these methods, additional processing time is necessary and theaccuracy of the restoration procedure is not guaranteed.

SUMMARY

A first embodiment according to the invention is a face identificationdevice, including a storage unit, a face detection unit, a feature valueobtainment unit, a reliability determination unit and a identificationunit. The storage unit stores a plurality of feature values previouslyobtained from a face image of a registrant. The storage unit may beconfigured to store feature values of one registrant or feature valuesof a plurality of registrants. The face detection unit detects the faceof a person from the input image. The feature value obtainment unitobtains a plurality of feature values from the detected face.

The reliability determination unit determines reliability of eachfeature value from the input image. Reliability is a value indicatinghow reliable the result obtained from identification processing is, whenidentification processing is performed by using the feature value. Thatis, if the subject in the input image faces sideways or wears a wornarticle such as sunglasses for example, it is difficult to accuratelyobtain specific feature values from the image. In such a case, if allfeature values are used with similar weighting in identificationprocessing, deterioration in identification accuracy may be caused.Therefore, the reliability determination unit decides reliability foreach feature value, and hands over the reliability to the identificationunit, which performs identification to prevent the identificationaccuracy from deteriorating. Note that a specific processing example ofthe reliability determination unit will be described later.

The identification unit identifies the detected face with the face ofthe registrant stored on the storage unit, by comparing the featurevalues stored on the storage unit with the feature values obtained fromthe detected face while taking into account the reliability of eachfeature value. That is, the identification unit performs identificationwhile performing weighting based on the reliability of each featurevalue.

In a face identification device configured as described above,identification is performed based on the reliability determined for eachfeature value. Therefore, even when a specific feature value cannot beobtained with high accuracy, it is possible to prevent theidentification accuracy from deteriorating by reducing the weighting ordisregarding the feature value when performing identification.

The face identification device according to a second embodiment of thepresent invention may be configured to further include a partial areadeciding unit for deciding a plurality of partial areas in an areaincluding the detected face. In such a case, the storage unit may storea plurality of feature values previously obtained for respective partialareas from the face image of the registrant. Further, in such a case,the feature value obtainment unit may obtain the feature value from eachpartial area of the input image. Additionally, the identification unitmay calculate a score of each partial area for the registrant bycomparing the feature value of the registrant stored on the storage unitwith the feature value obtained by the feature value obtainment unit foreach partial area, and based on the score and the reliability, mayidentify the detected face with the face of the registrant stored on thestorage unit.

The face identification device according to a third embodiment of thepresent invention may be configured to further include a directiondetermination unit for determining the direction of the detected face.In such a case, the reliability determination unit may determine thereliability of each feature value corresponding to the direction of thedetected face.

The face identification device according to a fourth embodiment of thepresent invention may be configured to further include a brightnessdetermination unit for determining the brightness of a part in whicheach feature value is to be obtained or the surrounding thereof. In sucha case, the reliability determination unit may determine the reliabilityof each feature value corresponding to the brightness.

The face identification device according to a fifth the presentinvention may be so configured as to further include a worn articledetermination unit for determining whether a worn article is includedwith the detected face. In such a case, the reliability determinationunit may determine the reliability of each feature value correspondingto whether a worn article being worn and a part where the worn articleis worn.

The face identification device according to a sixth embodiment of thepresent invention may be so configured as to further include a directiondetermination unit for determining the direction of the detected face.In such a case, the partial area deciding unit may decide a range ofeach partial area corresponding to the direction of the detected face.

Several embodiments of the present invention may be implemented as aprogram for causing processing performed by each unit described above tobe executed with respect to an information processor, or a recordingmedium on which the program is written. Further, a seventh embodimentaccording to the present invention may be specified as a method in whichprocessing performed by each unit described above is executed by aninformation processor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a function block examples of a face identification device;

FIG. 2 shows a specific example of partial areas;

FIGS. 3A to 3E show examples of face directions;

FIG. 4 shows an example of a reliability table corresponding to the facedirections of the subject;

FIGS. 5A to 5C show examples of effects of lighting environment;

FIG. 6 shows an example of a reliability table corresponding to theeffects of lighting environment;

FIGS. 7A to 7C show examples of effects of worn articles;

FIG. 8 shows an example of a reliability table corresponding to theeffects of the worn articles;

FIG. 9 shows a flowchart showing an exemplary operation of a faceidentification device; and

FIG. 10 shows an example of distortion corresponding to a facedirection.

DETAILED DESCRIPTION

FIG. 1 shows function block examples of a face identification device 1.The face identification device 1 obtains feature values (e.g.,brightness distribution and histogram) of a face from an image(identification image) in which a human who is the identificationsubject is captured, and compares them with the feature values of eachperson having been registered to determine who the person in theidentification image is. The face identification device 1 includes, ashardware, a CPU (Central Processing Unit) connected via a bus, a mainmemory (RAM) and an auxiliary memory.

The face identification device 1 includes an image input block 2, animage storage block 3, a face detection block 4, a feature pointdetection block 5, a state determination block 6, a partial areadeciding block 7, a reliability storage block 8, a reliabilitydetermination block 9, a registered information storage block 10 and aface identification block 11. Various programs (OS, applications, etc.)stored on the auxiliary memory are loaded to the main memory andexecuted by the CPU. The face detection block 4, the feature pointdetection block 5, the state determination block 6, the partial areadeciding block 7, the reliability determination block 9 and the faceidentification block 11 are realized when the programs are executed bythe CPU. Further, the face detection block 4, the feature pointdetection block 5, the state determination block 6, the partial areadeciding block 7, the reliability determination block 9 and the faceidentification block 11 may be configured as a dedicated chip. Next,respective functional blocks included in the face identification device1 will be described.

The image input block 2 works as an interface for inputting data of aidentification image to the face identification device 1. By the imageinput block 2, data of a identification image is input to the faceidentification device 1. The image input block 2 may be configured byusing any existing technique for inputting data of a identificationimage to the face identification device.

For example, data of a identification image may be input to the faceidentification device 1 over a network (e.g., local area network or theInternet). In such a case, the image input block 2 is configured byusing a network interface. Further, data of a identification image maybe input to the face identification device 1 from a digital camera, ascanner, a personal computer, a recording device (e.g., hard disk drive)or the like. In such a case, the image input unit 2 is configuredcorresponding to a standard (e.g., standard for a wire connection suchas USB (Universal Serial Bus) or SCSI (Small Computer System Interface)or wireless connection such as Bluetooth (registered trademark)) forconnecting the face identification device 1 with a digital camera, ascanner, a personal computer, a recording device or the like in a mannerenabling data communications. Further, data of a identification imagewritten on a recording medium (e.g., any kind of flash memory, flexibledisk, CD (Compact Disk), DVD (Digital Versatile Disc, Digital VideoDisc)) may be input to the face identification device 1. In this case,the image input unit 2 is configured by using a device for reading outdata from the recording medium (e.g., flush memory reader, flexible discdrive, CD drive or DVD drive).

Further, the face identification device 1 may be incorporated in animage capturing apparatus such as a digital camera or any kind ofapparatus (e.g., PDA (Personal Digital Assistant) or mobile telephone),and data of a captured identification image may be input to the faceidentification device 1. In this case, the image input block 2 may beconfigured using a CCD (Charge-Coupled Device), a CMOS (ComplementaryMetal-Oxide Semiconductor) sensor or the like, or configured as aninterface for inputting data of a identification image captured by a CCDor a CMOS sensor to the face identification device 1. Moreover, the faceidentification device 1 may be incorporated in an image output devicesuch as a printer or a display, and image data input into the imageoutput device as output data is input into the face identificationdevice 1 as an identification image. In this case, the image input block2 is configured using a device for converting the image data input tothe image output device into data which can be handled in the faceidentification device 1.

Further, the image input block 2 may be configured to be able to copewith the multiple cases mentioned above.

The image storage block 3 is configured using a memory. As a memory usedas the image storage block 3, any specific technique may be applied suchas a volatile memory or a nonvolatile memory.

The image storage block 3 stores data of a identification image inputvia the image input block 2. Data of an identification image stored onthe image storage block 3 is read out by the face detection block 4, thefeature point detection block 5, and the partial area deciding block 7.The image storage block 3 holds data of the identification image, whichis the subject of the processing, until processing at least performed bythe face detection block 4, the feature point detection block 5, thestate determination block 6, the partial area deciding block 7 and theface identification block 11 has been completed.

The face detection block 4 reads data of a identification image from theimage storage block 3, and detects the human face from theidentification image, and specifies the position and the size of thedetected face. The face detection block 4 may be configured so as todetect the face by template matching using a reference templatecorresponding to the contour of the whole face, or configured so as todetect the face by template matching based on organs (eyes, nose, ears,etc.) of the face. Alternatively, the face detection block 4 may beconfigured so as to detect the vertex of the head or the like bychroma-key processing to thereby detect the face based on the vertex.Alternatively, the face detection block 4 may be configured so as todetect an area of a color similar to the skin color to thereby detectthe area as the face. Alternatively, the face detection block 4 may beconfigured so as to perform learning by teacher signals using a neuralnetwork to thereby detect a face-like area as the face. Further, facedetection processing by the face detection block 4 may be realized byapplying any other existing techniques.

Further, if faces of multiple people are detected from an identificationimage, the face detection block 4 may decide which face is to be thesubject of processing according to predetermined criteria. Predeterminedcriteria include face size, face direction, and face position in animage.

The feature point detection block 5 detects a plurality of featurepoints for a face detected by the face detection block 4. Any of theexisting feature point detection technique may be applied to the featurepoint detection block 5. For example, the feature point detection block5 may be configured so as to previously learn patterns showing thepositions of face feature points and perform matching using the learneddata to thereby detect feature points. Alternatively, the feature pointdetection block 5 may be configured so as to detect edges and performpattern matching inside the detected face to thereby detect end pointsof the organs of the face, and by using them as references, detectfeature points. Alternatively, the feature point detection block 5 maybe configured so as to previously define shapes or texture models suchas AAM (Active Appearance Model) and ASM (Active Shape Model), and bydeforming them corresponding to the detected face, detect featurepoints.

The state determination block 6 determines, relating to the facedetected by the face detection block 4, whether any unfavorablecondition for identification has been caused, and if caused, determinesthe level. For example, the state determination block 6 may determinethe angle of the direction of the detected face. Such a determinationcan be performed in the following manner, for example. First, the statedetermination block 6 obtains corresponding relationships between faceimages or parts thereof and face directions and correspondingrelationship between arrangements of feature points of the face and facedirections, for example, by learning them previously, to thereby be ableto know the angle of the direction of the corresponding face. Such adetermination of angle may be performed by applying another technique.

Further, the state determination block 6 may determine the effect of alighting environment. Such a determination can be performed in thefollowing manner for example. First, the state determination block 6obtains the brightness for each partial area decided by the partial areadeciding block 7 described later. Then, the state determination block 6determines for each partial area that to what degree it is dark orbright. Alternatively, the state determination block 6 may performprocessing in the following manner. First, the state determination block6 sets a point or an area corresponding to each partial area in thedetected face, and obtains the brightness for each of them. The statedetermination block 6 may be configured to determine darkness orbrightness for each partial area based on the obtained brightness.Further, determination of the effect of such a lighting environment maybe performed by applying another technique.

Further, the state determination block 6 may determine whether there isany worn article. Such a determination can be performed in the followingmanner, for example. First, the state determination block 6 previouslyperforms learning based on an image in which a worn article, such assunglasses or a mask, is worn. Then, the state determination block 6performs detection processing based on the learning in the face detectedby the face detection block 4, and if any worn article is detected,determines that the worn article is worn. Alternatively, the statedetermination block 6 may be configured so as to obtain templates andcolor information about worn articles and perform detection by obtainingthe correlation thereto. Alternatively, based on an empirical rule thatentropy and dispersion deteriorate if the subject wares a worn articlesuch as sunglasses or a mask, the state determination block 6 may beconfigured so as to detect by using a substitute index such as entropy.Further, if a worn article is detected, the state determination block 6may be configured so as to obtain a value (concealment ratio) to whatdegree the face is concealed by the worn article. The concealment ratiocan be obtained by examining to what degree the feature points of anorgan likely to be concealed can be detected near the part where theworn article is detected, for example.

Further, the state determination block 6 may be configured to obtain theinformation amount included in the image of each partial area, for eachpartial area decided by the partial area deciding block 7 describedlater. The information amount can be shown as entropy for example.Alternatively, the information amount may be shown as relative entropyor KL (Kullback Leibler) divergence of the same partial area in an idealimage having sufficient information amount.

The partial area deciding block 7 designates partial areas based onpositions of the plural feature points detected by the feature pointdetection block 5. A partial area is an area which is a part of a faceimage. For example, the partial area deciding block 7 may be configuredso as to designate a polygon such as a triangle having feature points atthe respective vertexes as a partial area.

FIG. 2 shows specific examples of partial areas. In FIG. 2, threepartial areas 101, 102 and 103 indicated by rectangles are designated bythe partial area deciding block 7. The partial area 101 is a partincluding the right eye and the surrounding thereof of the subject. Thepartial area 102 is a part including the left eye and the surroundingthereof of the subject. The partial area 103 is a part including thenose and the surrounding thereof of the subject. The partial areadeciding block 7 may be configured to designate partial areas differentfrom them, or to designate more partial areas.

The reliability storage block 8 is configured by using a storage device.The reliability storage block 8 stores a reliability table. Thereliability table contains reliability values for respective partialareas in various conditions. Reliability is a value showing to whatdegree the result obtained from identification processing is reliablewhen identification processing is performed using the feature value ofthe partial area. In the following description, the reliability value isindicated as a value in a range of 0 to 1, in which the reliability islower as the value becomes close to 0, and the reliability is higher asthe value becomes close to 1. Note that values of the reliability may becontinuous values or discrete values.

First, reliability corresponding to a face direction of the subject willbe described. FIGS. 3A to 3E show examples of face directions. As shownin FIGS. 3A to 3E, in the present embodiment, angle is expressed suchthat the right side for whom viewing the identification image indicatespositive value and the left side indicates negative value. FIGS. 3A to3E show examples of the face of the subject facing −60°, 30°, 0°, +30°and +60°, respectively. FIG. 4 shows a specific example of a reliabilitytable 8 a corresponding to the face directions of the subject. Thereliability table 8 a in FIG. 4 shows reliabilities in the respectivepartial areas 101 to 103 for the respective face directions (θ).

Explanation will be given exemplary for the case of −60°. Almost allpart of the left eye of the subject appears in the image, but the halfof the right eye is hidden. Therefore, compared with the reliability ofthe partial area 102 corresponding to the left eye, the reliability ofthe partial area 101 corresponding to the right eye is set substantiallylower. Further, in the case of −30°, the right eye appears in the imageto some extent, compared with the case of −60°. Therefore, for thereliability in the partial area 101, the case of −30° is set higher thanthe case of −60°. Further, the feature value used in identification(feature value stored on the registered information storage block 10described later) is generally a feature value obtained from a full-faceimage. Therefore, if the face of the subject faces sideways in theidentification image, the accuracy deteriorates due to distortions beingcaused or the like, irrespective of all feature points in the partialarea appearing on the image or not. Accordingly, the reliability of eachpartial area 101 to 103 is set lower as the angular absolute valueincreases.

Next, explanation will be given for a reliability table corresponding toeffects of lighting environments. FIGS. 5A to 5C show specific examplesof effects of lighting environments. In FIG. 5A, the whole face of thesubject is lighted uniformly. On the other hand, in FIG. 5B, the leftside of the face of the subject is lighted but the right side isshadowed. Therefore, in FIG. 5B, the surrounding part of the right eyeis shadowed, so it is difficult to obtain the accurate feature value inthe partial area 101. Further, in FIG. 5C, the right side of the face ofthe subject is lighted properly but the left side is lightedexcessively, causing halation. Therefore, in FIG. 5C, halation is causedin the part of the left eye, so it is difficult to obtain the accuratefeature value in the partial area 102.

FIG. 6 shows a specific example of a reliability table 8 b correspondingto the effects of lighting environments. In this example, relationshipbetween lighting environment and reliability is defined for all partialareas in common. In any partial area, reliability is decided dependingon how many percentages of the area being shadowed or caused halation(e.g., may be expressed by using concealment ratio).

Next, explanation will be given for a reliability table corresponding toeffects of worn articles. FIGS. 7A to 7C show specific examples of theeffects of worn articles. In FIG. 7A, the subject does not have a wornarticle on the face, so the features in all partial areas appear on theimage. On the other hand, in FIG. 7B, the subject wares sunglasses, sothe right and left eyes are concealed, making it difficult to obtain theaccurate feature value in the partial areas 101 and 102. Further, inFIG. 7C, the subject wares a mask, so the nose, mouth, cheeks etc., areconcealed, whereby it is difficult to obtain the accurate feature valuein the partial area 103.

FIG. 8 shows a specific example of a reliability table 8 c correspondingto the effects of worn articles. In this example, if sunglasses areworn, the reliability of the partial areas 101 and 102 is 0, and thereliability of the partial area 103, not affected by the sunglasses,is 1. On the other hand, if a mask is worn, the reliability of thepartial area 103 is 0, and the reliability of the partial area 101 and102, not affected by the mask, is 1. Values in the reliability table 8 care not necessarily limited to “0” and “1”, but may be expressed byusing other values (e.g., “0.1” and “0.9”).

Next, explanation will be given for a reliability table corresponding tothe information amount obtained for each partial area. In this case, thereliability table is configured so that the reliability becomes closerto 1 as the information amount increases, and the reliability becomescloser to 0 as the information amount decreases.

The reliability determination block 9 determines the reliability of eachpartial area decided by the partial area deciding block 7, based on thedetermination results by the state determination block 6 and thereliability tables stored on the reliability storage block 8. If it isdetermined by the state determination block 6 that a plurality ofadverse effects exist, the reliability determination block 9 may beconfigured so as to calculate statistics (average, center of gravity,etc.) of the reliabilities corresponding to the respective conditionsand decide them as the reliabilities. Further, priority may be set forthe respective conditions, and the reliability determination block 9 maybe configured so as to decide the reliability obtained for the conditionof the highest propriety as the reliability of the partial area.

The registered information storage block 10 is configured using astorage device. The registered information storage block 10 stores aregistered information table. The registered information table includesthe feature value of each partial area for each ID of a person who is tobe the subject of identification. In the registered information table,for the feature value corresponding to an ID “A” for example, thefeature value of the partial area 101, the feature value of the partialarea 102 and the feature value of the partial area 103 are associated.

The face identification block 11 performs identification processingbased on the feature values obtained from the respective partial areas,the feature values stored on the registered information storage block10, and the reliabilities decided by the reliability determination block9. Hereinafter, processing by the face identification block 11 will bedescribed specifically. In the following description, the number ofpartial areas is not limited to three but is assumed as N pieces.

First, the face identification block 11 calculates the final scores forall IDs stored on the registered information storage block 10. The finalscore is calculated by executing score calculation processing and finalscore calculation processing using the calculated scores.

In the score calculation processing, the face identification block 11calculates scores of respective partial areas (j=1, 2, . . . , N). Ascore is a value calculated based on the feature value obtained fromeach partial area, indicating the possibility that the subject havingthe partial area is a specific subject. For example, a large score isobtained if the possibility is high and a small score is obtained if thepossibility is low.

The score of each partial area can be calculated according to Equation 1for example. Equation 1 indicates the probability P(ψ_(j)|x_(j), θ) thatthe person (face) is the subject person when a feature value x_(j) in apartial area j is obtained under a condition θ. However, ξ is a hyperparameter, which is a probability variable indicating how much thefeature value obtained under the condition θ deviates from a featurevalue which should be obtained originally under an ideal condition(deviance). ψ is a probability variable indicating the ID of aregistrant. Although the deviance ξ is treated as a discrete variable inEquation 1, it may be a continuous variable. Further, although a commoncondition θ is used for all partial areas in Equation 1, a differentcondition θ_(j) may be used for each partial area of course.

$\begin{matrix}{{P\left( {{\psi_{j}\text{|}x_{j}},\theta} \right)} = {\sum\limits_{\xi}\;{{P\left( {{\psi_{j}\text{|}x_{j}},\xi} \right)}{P\left( {\xi\text{|}\theta} \right)}}}} & \left( {{Equation}\mspace{20mu} 1} \right)\end{matrix}$

P(ψ_(j)|x_(j), ξ) indicates a value (score not taking into account thecondition θ) which can be obtained by comparing the feature valuesobtained from partial areas of the image with the feature values storedon the registered information storage block 10. This value is, forexample, the similarity between both feature values. The similarity canbe obtained by applying a face identification technique conventionallyused. For example, normalized correlation of brightness distribution andhistogram intersection of color histogram can be obtained as similarity.

On the other hand, P(ξ|θ) indicates reliability decided by thereliability determination block 9. That is, it is a value indicatingreliability (or accuracy) of the feature value x_(j) obtained in apartial area j under the condition θ (or similarity calculated from thefeature value x_(j) thereof). The reliability takes a smaller value asthe condition θ becomes worse.

In other words, P(ψ_(j)|x_(j), θ) in Equation 1 can be a value (adjustedscore) obtained by adjusting the similarity P(ψ_(j)|x_(j), ξ) betweenthe feature value of the subject and the feature value of theregistrant, based on the state of the subject (reliability) P(ξ|θ). Inthe final score calculation processing described later, the final scoreused for final evaluation is calculated from the adjusted scores of therespective partial areas. For the final score, a greater effect is givento a score having high reliability. Therefore, the reliability P(ξ|θ)can be considered to be “weighting” when the final score is calculated.

When the score calculation processing has been completed, the faceidentification block 11 then performs the final score calculationprocessing. In the final score calculation processing, the faceidentification block 11 calculates the final score on the basis of thescores obtained from the respective partial areas. Equation 2 is anequation for calculating the final score in the case where therespective partial areas are considered as independent. Equation 3 is anequation for calculating the final score based on Bagging model orMixture of Experts model, from which the final score can be obtained asthe average of the scores of a local area. The face identification block11 calculates the final score by using Equation 2 or 3, for example.Equations 2 and 3 are specific examples of processing capable of beingapplied in calculating the final score. The final score may be obtainedby another processing. For example, the face identification block 11 maybe so configured as to obtain the maximum value of the scores obtainedfrom the respective partial areas as the final score.

$\begin{matrix}{{P\left( {{\psi\text{|}x},\theta} \right)} = {\prod\limits_{j}^{N}\;{P\left( {{\psi_{j}\text{|}x_{j}},\theta} \right)}}} & \left( {{Equation}\mspace{20mu} 2} \right) \\{{S\left( {{\psi\text{|}x},\theta} \right)} = {\frac{1}{N}{\sum\limits_{j}^{N}\;{P\left( {{\psi_{j}\text{|}x_{j}},\theta} \right)}}}} & \left( {{Equation}\mspace{20mu} 3} \right)\end{matrix}$

When the final scores are calculated for all IDs, the faceidentification block 11 determines who the subject in the identificationimage is, that is, whether the person has an ID or not. For example, ifthe largest final score exceeds a prescribed threshold, the faceidentification block 11 determines that the subject is a person with anID corresponding to the final score. On the other hand, if the largestfinal score does not exceed a prescribed threshold, the faceidentification block 11 determines that the subject in theidentification image is not a person with an ID stored on the registeredinformation storage block 10, that is, an unregistered person. Then, theface identification block 11 outputs the determination result as theresult of face identification.

FIG. 9 is a flowchart showing an operating example of the faceidentification device 1. Next, an operating example of the faceidentification device 1 will be explained. First, an identificationimage is input into the face identification device 1 via the image inputblock 2 (S01). The image storage block 3 stores data of the inputidentification image. The face detection block 4 detects the face of aperson from the identification image stored on the image storage block 3(S02). Next, the feature point detection block 5 detects a plurality offeature points from the detected face, and based on the feature points,the partial area deciding block 7 decides partial areas (S03).

Next, the state determination block 6 determines whether any unfavorablecondition is caused in the detected face, and if caused, determines thedegree thereof (S04). The reliability determination block 9 obtains thereliabilities of the respective partial areas based on the determinationresults by the state determination block 6 (S05).

Next, the face identification block 11 performs score calculationprocessing (S06) and final score calculation processing (S07) for eachID (each registrant) stored on the registered information storage block10. This processing is repeated until processing for all registrants arecompleted (S08-NO). When the processing is performed for all registrants(S08-YES), the face identification block 11 decides the identificationresult based on the final score obtained for each registrant (S09).Then, the face identification block 11 outputs the identificationresult.

In the identification processing by the face identification block 11,degrees of effects of the feature values according to the respectivepartial areas are determined based on the reliabilities, which preventsdeterioration in the accuracy of face identification processing in aidentification image in which an unfavorable condition is caused.

Further, since it is configured so that the reliability determined fromthe image is multiplied with the similarity (score), functions such asface detection, feature value extraction, and similarity calculation canbe performed commonly in all conditions. Therefore, it is possible toreduce the development cost, the program size, and used resources, whichis extremely advantageous in implementation.

The face identification block 11 may be so configured as to performscore normalization processing after the final scores for all IDs arecalculated. Score normalization processing can be performed in order tocollect the values in a certain range if the range of the obtained finalscores is too broad.

Further, if the face of the subject in the input identification imagedoes not face the front, partial areas decided by the partial areadeciding block 7 may be distorted depending on the direction. Further,the feature value used in identification by the face identificationblock 11 is generally a feature value obtained from a face image facingthe front. Therefore, a distortion caused by the direction may cause anerror in identification processing by the face identification block 11.FIG. 10 shows a specific example of a distortion corresponding to a facedirection. Conventionally, there is a case where the distance betweenthe both eyes is used in deciding partial areas. However, since thedistance between the eyes depends largely on the face direction, in sucha method of deciding partial areas, accuracy in identificationdeteriorates significantly according to the face direction of thesubject. In the case of FIG. 10, it is also found that the distancebetween the eyes (t1) when facing sideways and the distance between theeyes (t2) when facing the front are significantly different. When theface direction is detected by the state determination block 7, thepartial area deciding block 7 may change the size or the range of eachpartial area corresponding to the direction, that is, performnormalization corresponding to the direction. More specifically, whenthe face direction is determined by the state determination block 6, thepartial area deciding block 7 may perform processing such as affineconversion to the partial area corresponding to the direction. Further,the partial area deciding block 7 may be configured to prepare templatescorresponding to a plurality of face directions, and to designate thepartial area by using a template corresponding to the determineddirection. With such a configuration, it is possible to obtain a featurevalue under a condition closer to the feature value stored on theregistered information storage block 10 to thereby prevent the accuracyof face identification from deteriorating due to face directions.

Further, the face identification device 1 may be configured so as toenable processing to register feature values in the registeredinformation storage block 10 by inputting an image capturing the face ofregistrant (hereinafter referred to as registered images) into the faceidentification device 1, rather than inputting feature values into theface identification device 1. In such a case, the registered image inputvia the image input block 2 is stored on the image storage block 3.Then, through cooperation between the face detection block 4, thefeature point detection block 5, the partial area deciding block 7 andthe face identification block 11, feature values of the respective areasare obtained from the registered image, and are stored on the registeredinformation storage block 10. In this case, the face identificationdevice 1 may be configured such that the state determination block 6determines the presence or absence of an unfavorable condition and itslevel in the registered image, and if it is in a certain level,registration will not be performed. With this configuration, it ispossible to prevent feature values obtained under unfavorable conditionsfrom being stored on the registered information storage block 10.Further, since face identification based on the feature values of lowaccuracy will not be performed, it is possible to prevent deteriorationin the accuracy of the face identification processing thereafter.

In the embodiment described above, identification processing of “one tomany”, that is, to which ID's registrant face and the face of thesubject coincides, is performed. However, the face identification devicecan be applied to identification processing of “one to one” in which itis determined whether the face photographed is the subject's face.

What is claimed is:
 1. A face identification device comprising: astorage unit for storing a plurality of feature values previouslyobtained for respective partial areas from a face image of a registrant;a processor comprising: a face detection unit for detecting a face of aperson from an input image; a partial area designating unit fordesignating a plurality of partial areas of an area including the face,the partial areas corresponding to feature points of a detected faceincluding at least one of an eye and a nose; a state determination unitfor determining from the detected face whether an unfavorable conditionfor identification of the detected face has occurred and, if so,determining the degree of the unfavorable condition, wherein unfavorableconditions include at least one of the angle of the direction of thedetected face, the effect of a lighting environment of the detectedface, a worn article on the detected face, and the entropy of eachpartial area of the detected face; a feature value obtainment unit forobtaining a plurality of feature values from the detected face forrespective designated partial areas; a reliability determination unitfor determining reliability for each feature value of the respectivedesignated partial areas based on the results obtained by the statedetermination unit including the degree of unfavorable conditions, andpredetermined reliability tables stored on a reliability storage unit,the reliability storage tables containing reliability values forrespective designated partial areas under various degrees of unfavorableconditions; and an identification unit for identifying the detected facewith the face of the registrant stored on the storage unit by comparingthe plurality of feature values stored on the storage unit for therespective partial areas of the face image of the registrant with theplurality of feature values obtained from the designated partial areas,while taking into account the reliability of each feature value of thedesignated partial areas; wherein the identification unit calculates ascore of each partial area for the registrant by comparing the featurevalues of the plurality of partial areas of the registrant stored on thestorage unit with the feature values obtained by the feature valueobtainment unit for the designated partial areas and based on the scoreand the reliability of each of the partial areas, identifies thedetected face with the face of the registrant stored on the storageunit.
 2. The face identification device according to claim 1, whereinthe state determination unit comprises a direction determination unitfor determining a direction of the detected face, wherein thereliability determination unit determines the reliability of eachfeature value of the plurality of designated partial areas correspondingto the direction of the detected face.
 3. The face identification deviceaccording to claim 1, wherein the state determination unit comprises abrightness determination unit for determining brightness of a part whereeach feature value of the plurality of designated partial areas is to beobtained or a surrounding thereof, wherein the reliability determinationunit determines the reliability of each feature value of the pluralityof designated partial areas corresponding to the brightness.
 4. The faceidentification device according to claim 1, wherein the statedetermination unit comprises a worn article determination unit fordetermining whether an article is worn by a subject in the detectedface, wherein the reliability determination unit determines thereliability of each feature value of the plurality of designated areascorresponding to whether an article being worn and a part where thearticle is worn.
 5. The face identification device according to claim 1,wherein the state determination unit comprises a direction determinationunit for determining a direction of the detected face, wherein thepartial area deciding unit designates a range of each partial areacorresponding to the direction of the detected face.
 6. A faceidentification method comprising the steps of: detecting a face, in aninput image, using an information processor having a storage unit forstoring a plurality of feature values previously obtained for respectivepartial areas from a face image of a registrant; designating a pluralityof partial areas in an area including the detected face using theinformation processor; determining from the detected face whether anunfavorable condition for identification of the detected face hasoccurred and, if so, the degree of the unfavorable condition using theinformation processor, wherein unfavorable conditions include at leastone of the angle of the direction of the detected face, the effect of alighting environment of the detected face, a worn article on thedetected face, and the entropy of each partial area of the detected faceimage; obtaining a feature value from each partial area of the detectedface using the information processor; determining a separate reliabilityfor each feature value of each designated partial area, usinginformation on the detection of an unfavorable condition and its degree,using the information processor; calculating a score of each partialarea by comparing the feature value of the registrant stored on thestorage unit with the feature value obtained by the feature valueobtainment unit for each partial area using the information processor;and based on the score and the reliability of each feature value of eachdesignated partial area, identifying the detected face with the face ofthe registrant stored on the storage unit using the informationprocessor by comparing the plurality of feature values stored on thestorage unit with the plurality of feature values obtained from thedetected face, while taking into account the reliability of each featurevalue.
 7. A program stored on a non-transitory computer readable medium,that, when executed by an information processor having a storage unitfor storing a plurality of feature values previously obtained forrespective partial areas from a face image of a registrant, causes theinformation processor to execute the steps of: detecting a face of aperson in an input image; designating a plurality of partial areas in anarea including the detected face; determining from the detected facewhether an unfavorable condition for identification of the detected facehas occurred and, if so, the degree of the unfavorable condition,wherein unfavorable conditions include at least one of the angle of thedirection of the detected face, the effect of a lighting environment ofthe detected face, a worn article on the detected face, and the entropyof each partial area of the detected face image; obtaining a featurevalue from each partial area of the detected face; determining aseparate reliability for each feature value from each designated partialarea using information on the detection of an unfavorable condition andits degree; calculating a score of each partial area by comparing thefeature value of the registrant stored on the storage unit with thefeature value obtained for each partial area; and based on the score andreliability, identifying the detected face with the face of theregistrant stored on the storage unit by comparing the plurality offeature values stored on the storage unit with the plurality of featurevalues obtained from the detected face while taking into account thereliability of each feature value.
 8. The device of claim 1, wherein atleast one unfavorable condition is a worn article on the detected faceand determining the degree of the unfavorable condition comprises usingan entropy index.
 9. The device of claim 1, wherein at least oneunfavorable condition is a worn article on the detected face anddetermining the degree of the unfavorable condition comprises obtaininga concealment ratio value.
 10. The device of claim 9, wherein obtaininga concealment ratio value comprises determining to what degree thefeature values of an area likely to be concealed can be detected nearthe part where the worn article is detected.
 11. The method of claim 6,wherein at least one unfavorable condition is a worn article on thedetected face and determining the degree of the unfavorable conditioncomprises using an entropy index.
 12. The method of claim 6, wherein atleast one unfavorable condition is a worn article on the detected faceand determining the degree of the unfavorable condition comprisesobtaining a concealment ratio value.
 13. The method of claim 12, whereinobtaining a concealment ratio value comprises determining to what degreethe feature values of an area likely to be concealed can be detectednear the part where the worn article is detected.
 14. The program ofclaim 7, wherein at least one unfavorable condition is a worn article onthe detected face and determining the degree of the unfavorablecondition comprises using an entropy index.
 15. The program of claim 7,wherein at least one unfavorable condition is a worn article on thedetected face and determining the degree of the unfavorable conditioncomprises obtaining a concealment ratio value.
 16. The program of claim15, wherein obtaining a concealment ratio value comprises determining towhat degree the feature values of an area likely to be concealed can bedetected near the part where the worn article is detected.